Many different types of so-called "special-effects" can be created using digital imaging techniques. One such class of special-effects techniques involves inserting the foreground of one image into a different background image. This makes it possible for a person or object to appear to be in a different setting than they really are. For example, the weatherman can appear to be standing in front of a weather map, when in reality he is standing in front of a blue wall, or an actor can appear to be standing on the edge of a cliff, when in reality he is actually performing in the safety of a studio. Typically, these methods rely on having the foreground object photographed in front of a brightly colored backdrop of a known color. A common backdrop color is blue, which is why this technique is often referred to as "blue-screening."
The basic steps involved with implementing a typical blue-screening algorithm are illustrated in FIG. 1. First, an object is photographed in front of a brightly colored backdrop of a known color which is shown as an image capture step 10. The captured image will contain a foreground region corresponding to the object being photographed, and a key color region, corresponding to the brightly colored backdrop. The key color region has a key color such as bright green or bright blue.
A segmentation step 12 is next used to segment the captured image into the foreground region and the key color region by detecting portions of the image that have the key color. Since the color of the backdrop will not be perfectly constant, the key color will typically be characterized by a range of color values surrounding some nominal color value, rather than a single point in color space.
Many blue-screening algorithms also include a control signal creation step 14. This is useful because the image will typically contain some foreground pixels that have been contaminated by the key color. For example, the pixels that occur along the boundary between the foreground region and the key color region usually contain a mixture of foreground color and key color. The control signal is determined to indicate the relative amount of foreground color and key color contained in each contaminated pixel.
Finally, an image composition step 16 is used to combine the foreground region of the captured image with a second background image. During this step, the foreground region of the captured image is inserted into the background image. For the foreground pixels that were determined to be contaminated with the key color, the control signal can be used to remove the appropriate amount of key color and replace it with the corresponding background image.
Several methods have been disclosed in the prior art for the image composition step 16 shown in FIG. 1. These methods generally involve the use of a control signal to regulate a fractional amount of the pixel color values that will be blended together from the first digital image and the second background digital image. Typically, these type of methods take the form of the equation: EQU R=(1-k)F+kS (1)
where R is the color value of the combined image, k is the control signal, S is the second background digital image, and F is the first digital image. Examples of these methods that can be found in prior art are shown in FIG. 2. See U.S. Pat. No. 5,381,184. FIG. 2(a) illustrates a threshold where the key color region 20 has a control signal value, k, equal to 1. The control signal is equal to 0 outside of the key color region 20. FIG. 2(b) illustrates a large key color region 22 where the control signal value k=1. Once again, all color values outside of the key color region 22 have a control signal value equal to 0. FIG. 2(c) illustrates a small key color region 24 where the control signal value k=1 and the area outside of the key color region has a control signal value k=0. FIG. 2(a)-2(c) all exhibit a binary control signal. These approaches will produce inferior composite images in that either there will be significant fringe effects around the foreground region in the composite image or that the foreground region will have been replaced with the new background around the periphery of what should have been the foreground region.
An improvement relative to the binary control signal methods can be obtained by making the control signal transitional such that the control signal varies from a first value, for example 0, to a second value, 1. This approach has become known as "soft chroma keying" and has enabled the blending of the first digital image with the second background digital image to have a smooth transition from the foreground region to the new background region. FIG. 2(d) illustrates a transitional control signal for a soft chroma keying method. The small dark shaded circle represents the key color region 28 and the large lightly shaded region represents the mixed region 26. The control signal value k ranges from 1 in the key color region 28 to 0 at the outer edge of the circle representing the mixed region 26. The control signal value falls smoothly from 1 to 0 when traversing from the outer edge of the key color region 28 to the outer edge of the mixed region 26. The soft chroma keying approach was an improvement to the previously described methods, however, there were still several problems with this method. An example of one problem with soft chroma keying involves how large the mixed region should be. If the mixed region extends too far out from the key color region then the background image will replace too much of the foreground region of the first image. On the other hand, if the mixed region doesn't extend far enough, then fringe effects will occur on the outer edge of the foreground region in the combined image where the foreground region is contaminated by the key color.
A further improvement to the above methods involved the use of a different image composition equation. Traditionally, Eq. (1) was used to combine the two images. However, when using this approach fringe effects could still be observed where the edges of the foreground region of the combined image were contaminated by the key color. This can be improved by the removal of some fraction of the key color from pixels contained in the mixed region. The amount of key color removed was based upon the control signal as shown in the following equation: EQU R=F+kS-kC (2)
where k is the control signal, F is the color value of a pixel in the first digital image, S is the color value for a corresponding pixel in the background digital image, C is the key color, and R is the resulting color value of the combined image. It can be seen that the effect of applying this equation to the contaminated pixels is to subtract a portion of the key color from the first digital image, and replace it with an equivalent portion of the second background digital image.
Despite the improvement realized using the approach given in Eq. (2), a number of deficiencies remain. In particular, all methods as described in the prior art are incapable of handling key color contamination from secondary illumination. In this context, secondary illumination is defined as the light reflecting off of the key color backdrop onto the subject causing contamination of foreground pixels with the key color. Generally, this contamination will be most obvious when portions of the subject that are away from the boundary between the foreground region (subject) and the key color region (backdrop) are effected. If the method of Eq. (2) is used to compensate for the key color contamination due to secondary illumination, incorrect results will be obtained. In this case, the key color contamination will be removed according to the control signal, and replaced with the new background. However, if the foreground pixels that are contaminated by the secondary illumination are a sufficient distance away from the boundary between the key color region and the foreground region, replacing the key color contamination with the new background can cause the foreground region to appear to be partially transparent. This has the effect of contaminating the foreground image with the background image, which can be just as objectionable as contamination by the key color.
Finally, another issue that the prior art methods have difficulty with is when the subject contains color values similar to those of the key color. In this case, the subject region will be classified as mixed pixels, and will therefore have intermediate control signal values. Therefore, a portion of the foreground image will be replaced by the background image during the image composition process. This will again have the effect of causing the foreground region to appear to be partially transparent.